[flang] Refine tokenization trick that hid macro name #121990
Conversation
In order to properly expose the Hollerith editing item in
something like FORMAT(3I9HHOLLERITH) as its own token, the
tokenization routine in the prescanner has special handling
for digit strings followed by letters ("3I" above). This
handler's effects are too broad, and prevent a macro name
from being recognized as such in a reported bug; make the
test for a hidden Hollerith more precise.
Fixes llvm#121931.
llvmbot
added
flang
|
@llvm/pr-subscribers-flang-parser Author: Peter Klausler (klausler) ChangesIn order to properly expose the Hollerith editing item in something like FORMAT(3I9HHOLLERITH) as its own token, the tokenization routine in the prescanner has special handling for digit strings followed by letters ("3I" above). This handler's effects are too broad, and prevent a macro name from being recognized as such in a reported bug; make the test for a hidden Hollerith more precise. Fixes #121931. Full diff: https://github.com/llvm/llvm-project/pull/121990.diff 2 Files Affected:
diff --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 3cd32d7e6c92e8..9cf093b012ccd8 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -709,9 +709,22 @@ bool Prescanner::NextToken(TokenSequence &tokens) {
QuotedCharacterLiteral(tokens, start);
} else if (IsLetter(*at_) && !preventHollerith_ &&
parenthesisNesting_ > 0) {
- // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
- // we don't misrecognize I9HOLLERITH as an identifier in the next case.
- EmitCharAndAdvance(tokens, *at_);
+ const char *p{at_};
+ int digits{0};
+ for (;; ++digits) {
+ ++p;
+ if (InFixedFormSource()) {
+ p = SkipWhiteSpace(p);
+ }
+ if (!IsDecimalDigit(*p)) {
+ break;
+ }
+ }
+ if (digits > 0 && (*p == 'h' || *p == 'H')) {
+ // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
+ // we don't misrecognize I9HOLLERITH as an identifier in the next case.
+ EmitCharAndAdvance(tokens, *at_);
+ }
}
preventHollerith_ = false;
} else if (*at_ == '.') {
diff --git a/flang/test/Preprocessing/bug129131.F b/flang/test/Preprocessing/bug129131.F
new file mode 100644
index 00000000000000..5b1a914a2c9e35
--- /dev/null
+++ b/flang/test/Preprocessing/bug129131.F
@@ -0,0 +1,5 @@
+! RUN: %flang -fc1 -fdebug-unparse %s 2>&1 | FileCheck %s
+! CHECK: PRINT *, 2_4
+#define a ,3
+ print *, mod(5 a)
+ end
|
| for (;; ++digits) { | ||
| ++p; | ||
| if (InFixedFormSource()) { | ||
| p = SkipWhiteSpace(p); |
There was a problem hiding this comment.
Could p possibly walk off the end of the buffer here?
There was a problem hiding this comment.
(Ghithub had issues. My comment was meant for line 715 ++p.)
There was a problem hiding this comment.
I don't think so; source files are normalized so that their last lines always end with a newline.
3 participants
In order to properly expose the Hollerith editing item in something like FORMAT(3I9HHOLLERITH) as its own token, the tokenization routine in the prescanner has special handling for digit strings followed by letters ("3I" above). This handler's effects are too broad, and prevent a macro name from being recognized as such in a reported bug; make the test for a hidden Hollerith more precise.
Fixes #121931.